Vigilance monitoring system

ABSTRACT

The present invention pertains to a system and method of monitoring the alertness or wakefulness of a driver. The monitored parameters include cardiac, respiratory and movement parameters. Sensors are located in various locations of the driver side section to detect the vigilance of a driver. These sensors include pressure sensors embedded in the seat and pedals, and a head band for monitoring EEG, EMG and EOG signals.

[0001] The present invention relates to a vigilance monitoring system.In particular the invention relates to a system for monitoring,recording and/or analysing vigilance, alertness or wakefulness and/or astressed state of an operator of equipment or machinery in a variety ofsituations including situations wherein the degree of vigilance of theoperator has implications for the safety or well being of the operatoror other persons. A typical application may include monitoring thedriver of a vehicle or pilot of an aircraft, although the invention alsohas applications in areas involving related occupations such as traindrivers and operators of equipment such as cranes and industrialmachinery in general, and where lack of operator vigilance can give riseto harmful social or economic consequences.

[0002] The system of the present invention will be described herein withreference to monitoring a driver of a vehicle nevertheless it is notthereby limited to such applications. For example, other applicationsmay include monitoring routine, acute or sub-acute physiologicalparameters of a person or subject in a home, work, clinic or hospitalenvironment. The monitored parameters may include cardiac, respiratoryand movement parameters as well as parameters relating to apnea events,subject sleep states or sudden death syndrome on-set.

[0003] The monitoring system is designed, inter alia, to providenon-invasive monitoring of a driver's physiological data includingmovement activity, heart activity, respiration and other physiologicalfunctions. The monitored physiological data may undergo specificanalysis processing to assist in determining of the driver's state ofvigilance. The system is designed to detect various states of thedriver's activity and detect certain conditions of driver fatigue orrelaxation state that could lead to an unsafe driving condition orconditions.

[0004] The system of the present invention includes means for gatheringmovement data associated with the driver. The movement gathering meansmay include a plurality of sensors such as touch sensitive mats placedin locations of the vehicle that make contact with the driver, such asthe seat, steering wheel, pedal(s), seat belt or the like. Each locationmay include several sensors or mats to more accurately monitor movementsof the driver.

[0005] Signals from the various sensors/mats may be processed andanalysed by a processing means. The processing means may include adigital computer. The processing means may be programmed to recognizeparticular movement signatures or patterns of movement, driver postureor profile and to interpret these to indicate that vigilance hasdeteriorated or is below an acceptable threshold. The processing meansmay include one or more algorithms.

[0006] The sensors or mats may include piezoelectric, electrostatic,piezo ceramic or strain gauge material. The latter may be manufacturedby separating two conductive materials such as aluminium foil with anelectrolyte material which is capable of passing AC but not DC current.In one form the sensors or mats may include Capacitive Static Discharge(CSD) or Polyvinylidene fluoride (PVDF) material. The sensors/mats maybe covered with a non-obtrusive, flexible surface which is capable ofdetecting pressure and/or monitoring electrophysiological activity.

[0007] The pressure detecting capability may be used for detectingdriver movement. The or each sensor may produce an output signal thatrepresents the magnitude of the pressure or force that is applied to thesensor. The or each pressure signal may thus represent an absolute orquantitative measure of pressure applied to the sensor. Theelectrophysiological activity may include electrical signals generatedby the body of the driver eg. electrical muscle activity and/or pulseactivity.

[0008] The sensors or mats may be located in various parts of a vehicle.The seat of the driver may be divided into several sections such asupper or back and lower or seat. The upper or back section may includesensors in the top edge. centre and base. The lower or seat section mayinclude sensors in the front edge, centre and rear. The or each sensormay include CSD or PVDF material.

[0009] The steering wheel may include a plurality of sensors. Thesteering wheel may be divided into eight zones such as upper, upperleft, upper right, left, right, lower left, lower right and lower. Atleast one sensor may be associated with each zone. The or each sensormay include CSD or PVDF material.

[0010] The floor covering such as carpet may include a plurality ofsensors. The floor covering or carpet may be divided into a plurality ofzones. At least one sensor may be associated with each zone. The or eachsensor may include CSD or PVDF material.

[0011] The accelerator, clutch and brake pedals may include a pluralityof sensors. Each pedal may be divided into a plurality of zones such asupper, middle and lower. At least one sensor may be associated with eachzone. The or each sensor may include CSD, PVDF or other movementsensitive material.

[0012] The seat belt may include one or a plurality of sensors. In oneform a sensor or sensors may be embedded in the fixed (i.e.non-retractable) section of the seat belt. The or each sensor mayinclude CSD or PVDF material.

[0013] In some embodiments a head tilt device incorporating a positionalswitch or the like may be associated with the drivers cap, glasses orgoggles or may be arranged to clip over the drivers ear or glasses. Thehead tilt device may be adapted to provide a signal or data which altersin accordance with the position of the driver's head. Alternatively aradio tracking device may determine and track a subject's headmovements.

[0014] In critical applications of vigilance monitoring includingapplications involving pilots of aircraft, persons responsible fornavigating/controlling shipping and drivers of road or rail transport itmay be desirable to utilize more comprehensive methods of vigilancemonitoring. The latter may include techniques used in conventional sleepmonitoring. A head band and/or chin band sensor may be used to monitorEEG, EMG and EOG signals. The head band sensor may include separate leftand right frontal zones and left and right eye zones. The sensor mayinclude CSD or PVDF material or other material sensitive to measuringpatient skin electrical surface variations and/or impedance.

[0015] Various sensors/techniques may be adapted for monitoring eyemovement including those based on reflected light, electric skinpotential, contact lenses, limbus tracking, video imaging and magneticinduction. The sensors/techniques may include EOG electrodes, infrareddetection of eye movements and/or video tracking and processing of eyemovements. The sensors/techniques may be adapted for monitoring the lefteye only or the right eye only or both eyes.

[0016] Raw data which is collected from the various sensors positionedaround the vehicle may be filtered and amplified prior to processing andanalysis. A significant purpose of the processing and analysis is todetermine the driver's state of vigilance, alertness or wakefulness. Insome embodiments, the system may be adapted to effect remedial action,ie. the system may take steps to alert the driver or to activelyintervene in the control of the vehicle, when it is deemed that suchaction is warranted or desirable.

[0017] Processing of data may be performed in several stages, includingprimary, secondary and tertiary analysis.

[0018] Primary analysis refers to processing of raw data from thevarious sensors. This raw data may be filtered and amplified prior toanalog to digital conversion. Primary analysis may be adapted todetermine valid body movements of the driver as distinct from spurioussignals and artefacts due to environmental factors including noise.

[0019] Valid body movements may be determined by applying a combinationof processing techniques including:

[0020] 1. signal threshold detection whereby signals below or above apredetermined threshold are ignored and/or classified as noise orartefact,

[0021] 2. frequency filtering whereby high-pass, low-pass and notchfilters are adapted to remove noise and artefact signals,

[0022] 3. signal compression whereby data is minimized by presentingmain data points such as signal peaks, troughs, averages and zerocrossings,

[0023] 4. half period, amplitude analysis of signals, including analysisas disclosed in AU Patent 632932 entitled “Analysis System forPhysiological Variables”, assigned to the present applicant, thedisclosure of which is incorporated herein by cross reference.

[0024] Threshold detection may facilitate distinguishing random andnon-significant electrical noise (typically spikes of small duration)relative to signals representing valid or actual body movements.Threshold detection may apply to both amplitude and duration of thesignals. The relevant threshold(s) may be determined from clinicaltrials and/or historical data. Where the detection is based on amplitudeit may be determined in both negative and positive phases of the signal.Amplitude detection may be based on a measurement of the peak-to-peaksignal. Alternatively, the positive and negative peak amplitudes can bemeasured separately. Threshold detection may be combined with a form ofzero-base line detection so that electronic offsets do not adverselyaffect the accuracy of threshold detections. Each body movement whichexceeds the predetermined amplitude and/or duration may be classified asan event for further processing.

[0025] Secondary analysis may be adapted to process the results ofprimary analysis. Secondary analysis may process data for the purpose ofpresentation and/or display. Data may be displayed or printed in atabular, graphical or other format which facilitates interpretation ofthe data. One purpose of the representation and/or display is torepresent a driver's state of vigilance and/or fatigue. In one form eachevent identified during primary analysis may be counted for a fixedperiod of time or epoch. The fixed period of time may be 30 seconds or60 seconds, or other period which is adequate for determining bodymovement trends. The count value or number of valid body movements in agiven period (eg. 30 seconds) may be represented in a display as, say,the length of a vertical bar.

[0026] Where it is desired to display the energy or power associatedwith valid body movements in a particular epoch or time period, theaverage amplitude associated with each event may be indicated by thelength of the vertical bar whilst the count value or number of validbody movements for each epoch may be represented by colouring thevertical bar. For example the colours green, blue, yellow, orange, redmay indicate count values or movement numbers in ascending order ie.green indicating the lowest movement number for a particular epoch andred indicating the highest movement number for a particular epoch.Alternatively, data may be displayed on a 3 dimensional graph whereinfor example the x dimension of the graph represents time or epochs, they dimension represents the average amplitude, while the z dimensionrepresents the number of events during a particular epoch. The abovedisplay techniques may facilitate interpretation of the number of validbody movements and the amplitude of those movements and association ofthis data with the driver's activity or state of vigilance, alertness orwakefulness.

[0027] It may also be relevant to measure the period of each individualbody movement as this may provide an indication of the energy that isassociated with the movement. For example, if a driver squeezes thesteering wheel in a rapid response as distinct from gripping the wheelas part of a focussed steering manoeuvre, the pattern of signals in eachcase will be different. The rapid response may appear as a small clusterof movements/signals or as a single movement/signal with a relativelyshort duration or period of time. In contrast, the steering manoeuvremay appear as a larger cluster of movements/signals over a relativelylonger period of time or as a single movement/signal having a relativelylong duration.

[0028] The type of signal which may be expected (cluster or singlemovement/signal) will depend in part upon the type of sensor. Forexample, piezo ceramic or PVDF sensors may emit fewer clusters ofsignals but may emit signals with larger time periods in relation to theactual period of the movement which is being monitored. A capacitiveelectrostatic sensor is more likely to emit clusters of “spikes” beingrelatively short period signals. It may be necessary to record theenergy level of each movement as this energy level may fall below acertain threshold when the driver is in a fatigued state. If, forexample the driver has relaxed, then the energy of a body movement inthe actions of driving may be significantly more subdued than in thecase where the driver is alert, and his muscle activity is significantlygreater. Therefore it may be useful to measure and record each and everybody movement. This data could be displayed on high-resolution graphswhere for example the X-axis represents ½ second periods and 960 linesmake up each continuous section- or 480 seconds (8 minutes). The largestamplitude signal in each ½ second period could then be displayed on theX-Axis. The Y-Axis on the other hand could be represented by a scale ofamplitudes representing each body movement. This graph would be moreprecise in representing the actual signal level of each body-movementand the subsequent muscle status for a driver.

[0029] It may also be useful to detect events that are represented bygroups of movements, where, for example, the groups of movements may beindicative of a driver activity of interest. Detection of groups ofmovements may include user configurable or preset values for;

[0030] the maximum time between consecutive body-movements in order toqualify as being counted as part of a periodic body-movement.

[0031] the number of consecutive body-movements that are required toqualify for a periodic movement.

[0032] the time period during which this number of body-movements mustexist in order to qualify as a periodic body-movement.

[0033] Periodic measurement analysis can detect, for example, absence ofmovements which can be associated with a driver's fatigue.

[0034] Tertiary analysis may be adapted to process the results ofsecondary analysis. One purpose of tertiary analysis is to determine thestatus of a drivers state of vigilance, alertness or wakefulness.Tertiary analysis may process the results of secondary analysis toproduce intermediate data and/or indicate trends in the data. Theintermediate data and trends may be used to provide summary reports andfurther tabular and/or graphic representations of a drivers status orcondition. The intermediate data may be processed by one or morevigilance algorithms to determine the status of a driver's vigilance,alertness or wakefulness. Intermediate data of various types may bederived and the vigilance algorithm(s) may make use of such data todetermine the status of the driver. The intermediate data may include:

[0035] Rate of change of body movement detections

[0036] Rate of change of body movement amplitudes

[0037] Area under curve of time versus body movement, for varioussequential epochs to detect trends of subject movement changes(amplitude or number of movements)

[0038] Correlation of sensor data for patterns of amplitude, energy andbody movement changes that can be associated with driver fatigue

[0039] Change in frequency of body movement signals

[0040] Change in amplitude periods of body movement signals

[0041] Change in phase relationships of body movement signals

[0042] Relative phase relationship between each section and other typesof sensor sections.

[0043] Following tertiary analysis the vigilance algorithm(s) may beadapted to correlate the intermediate data and/or apply combinationallogic to the data to detect patterns of movement (or lack thereof)which, based on historical data or clinical trials, indicates that thedriver is or may be excessively relaxed or is below an acceptablethreshold of vigilance, alertness or wakefulness.

[0044] The vigilance algorithm(s) may incorporate one or more look uptables including reference movement data and default values associatedwith acceptable and unacceptable levels of driver fatigue. Histogramsincluding movement histograms of the kind described in AU Patent 632932based on the work of Rechitschaffen and Kayles (R & K) may be used aswell as tables showing weighted values and actual movement data for eachsensor.

[0045] The vigilance algorithm(s) may determine a vigilance probabilityfactor (0-100%) as a function of weighted movement data values.

[0046] Upon detecting a vigilance probability factor which is below anacceptable threshold, the system may be arranged to intervene in thecontrol of the vehicle or to alert the driver of the vehicle and/orother vehicles. Vehicle control intervention may include restriction ofspeed, controlled application of brakes, cutting-off fuel and/ordisabling the accelerator pedal. Driver alerting intervention mayinclude use of sprays designed to stimulate the driver, vibrating thesteering wheel, seat belt or floor area in the vicinity of the driver,an audible alarm and/or use of bright cabin lights. The driver can alsobe alerted by winding down the driver window and/or other effectivealerting methods as may be applicable to each individual driver. Driversof other vehicles may also be alerted by means of flashing hazard lightsand/or sounding of a siren. Vehicle control intervention may beintegrated with and form part of a vehicle control system or it may beinterfaced to an existing vehicle control system. Vehicle controlintervention may be interfaced with GSM or other communication systemsto provide early warning indication that a driver or operator ofequipment is in a stressed, fatigued or other undesirable condition thatmay be detected.

[0047] To assist differentiating normal and acceptable driver vigilancefrom fatigued or inappropriate driver conditions, calibration of thevarious sensor and transducer outputs is possible. Calibration can setthe system's detection parameters in accordance with varying drivermovement and other driver signals. Calibration is beneficial becausedriver sensor and signal outputs will vary with different drivers.Background noise will also vary with different vehicles. The need forcalibration may be proportional to the critical nature of the driving ordependent on the level of accuracy required for fatigue monitoring anddetection.

[0048] The need for calibration may to some extent be removed byutilizing artificial intelligence to distinguish baseline conditions fora drivers normal wakeful state to facilitate subsequent analysis anddetermining when a driver's state indicates fatigue or lapse ofvigilance. Artificial intelligence may be embodied in one or moreautomated systems including one or more mathematical algorithms.Artificial intelligence includes the systems ability to self-learn orteach itself conditions associated with the driver which constitutenormal or alert driving as distinct from conditions which constituteabnormal or fatigued driving.

[0049] Artificial intelligence may allow the driver of a specificvehicle to select a mode of operation during which the driver'smovements during normal or wakeful driving are monitored and diagnosedin order to determine typical thresholds and correlations betweenvarious sensors, for the purpose of determining true fatigue states ofthe driver as distinct from alert states of the driver. Artificialintelligence may also facilitate adaptation of the vigilancealgorithm(s), to the specific vehicle's background noisecharacteristics.

[0050] Artificial intelligence may include different response patternsfor correlating movement data from the various sensors fordistinguishing valid driver movements from environmental vibrations andnoise. These may be classified and described by, for example, a look uptable that records expected patterns or combinations of signals fordifferent cases of environmental noise as distinct from driver generatedsignals. For example, if the driver moves his hand, signals from sensorsin the steering wheel and arm sections of the seat may correlateaccording to a specific pattern. Alternatively, if the vehicle undergoesa severe or even subtle vibration due to road or engine effects, abroader range of sensors may be similarly affected and this may bemanifested as amplitudes which follow predetermined correlationpatterns. Signals from the sensors may increase in strength or amplitudeaccording to the proximity of the source of the sound or vibrationsWhere the source of the vibration is localized, this may manifest itselfas a pattern of similar waveforms across the various sensors whichreduce progressively in amplitude as the sensor's distance from thesource increases. For example, if the source of the vibration is roadnoise, the floor sensors may register maximum amplitude whereas thesteering wheel sensors which are furthest from the road noise mayregister minimum amplitude.

[0051] The phase relationship of vibrations from various sources mayalso provide some guide as to the likely source of the vibrations. Forexample, if the vibrations emanate from the driver's movement then it ismore likely that several signals with similar phase may be detected. Onthe other hand, if the signals have varying phase relationships, then itis more likely that the source of the vibrations giving rise to thesesignals is random as may be expected if the vibrations emanate from thevehicle environment.

[0052] Similar phase signals arising from driver movements may bedistinguished from similar phase signals arising from artefacts or thevehicle environment by relating the environmental noise to sensorslocated near sources of expected noise in the vehicles, eg. enginenoise, wheel noise, and other vibrations and noise. This may be detectedby carefully locating microphones and vibration sensors in the vehicle.

[0053] Cancellation of environmental noise can be assisted by monitoringsignals from the microphones and sensors with a view to applying themost effective signal cancellation techniques in order to reduce as muchas possible the artefact or noise effects or unwanted signals within thevehicle environment.

[0054] One example of the application of noise cancellation techniquesincludes detection of the various road bumps and ignoring the effect ofthese bumps on the data being analysed from the various vehicle sensorsof interest.

[0055] Another example of the application of noise cancellationtechniques includes detection of various engine noises and applicationof a signal of opposite phase to the motor noise in order to cancel theartefact. One example of phase cancellation techniques which may beadopted is disclosed in PCT application AU97100278, the disclosure ofwhich is incorporated herein by cross-reference.

[0056] Other examples of noise cancellation include filtering whereinhighpass, lowpass and notch filters may be used to assist artefactremoval.

[0057] Artificial intelligence may learn to ignore periodic signals fromsensors in the vehicle as these are likely to arise from mechanicalrotations within the vehicle, thus improving the separation of artefactsignals from signals of interest, such as signals which indicate truedriver movement.

[0058] Artificial intelligence may also learn to recognize changes inthe driver's state which reflect changes in driver vigilance orwakefulness. Points of calculation and analysis of sensor data for thepurpose of comparison and correlation with previously monitored data mayinclude:

[0059] spectral analysis of signals with a range of consecutive timeperiods;

[0060] ½ period time amplitude analysis of signals and other techniquesused in conventional sleep analysis as disclosed in AU Patent 632932;

[0061] calculation of the number of movements per consecutive periods oftime, wherein the consecutive periods of time may typically be, 1 secondor ½ second;

[0062] calculation of average signal levels during periods of, say, 20or 30 seconds;

[0063] calculation of total “area under the curve” or integration ofsensor signals for a period of, say, 20 or 30 seconds;

[0064] correlation and relationship between various combinations ofinput sensor channels;

[0065] ECG heart rate and respiration signals, the latter signalsproviding an indication of the driver's wakeful state, as heart-rate andrespiration signals during the sleep state are well documented in anumber of medical journals.

[0066] Artificial intelligence may be applied in conjunction withpressure sensors in vehicle seats and/or seat belts to control air bagdeployment. In this way air bag deployment may be restricted forchildren or validated with different crash conditions for children andadults. For example, if a child is not detected as being thrust forwardby means of pressure data received from seat/seat belt sensors,deployment of air bags and possible air bag injury to the child may beavoided.

[0067] Deployment of air bags may generally be validated moreintelligently by analysing data relating to passenger or driver posture,movement, thrust, body movement, unique driver or passenger ‘yaw’ etc.

[0068] The system may include means for testing a driver's responsetimes. Such tests may, if carried out at regular intervals, pre-emptserious driving conditions as can be brought about by driver fatigue ora lapse in vigilance. The testing means may be adapted to provide asimple method for prompting the driver and for testing the driver'sresponse time. The response test may, for example, request the driver torespond to a series of prompts. These prompts may include requesting thedriver to squeeze left or right hand sections of the steering wheel orsqueeze with both left and right hands at the same time in response to aprompt. The means for prompting the driver may include, for example,LEDs located on the dash of the vehicle or other position that thedriver is visually aware of. A left LED blinking may for example, promptthe driver to squeeze the left hand on the steering wheel. A right LEDblinking may prompt the driver to squeeze the right hand on the steeringwheel. The centre LED blinking may prompt the driver to squeeze bothhands on the steering wheel. Alternatively two LEDs could be used in theabove example, except that both LEDs blinking may prompt the driver tosqueeze with both hands.

[0069] The drivers response or level of alertness may be detected bymeasuring the response time of the driver, where the response time ismeasured as the time between illumination of an LED and a correctresponse with the hand or hands. In a case where an inappropriateresponse time is detected (potentially signalling driver fatigue oronset of driver fatigue) the system can verify the results and alert thedriver. The system may also determine the accuracy of the driver'sresponses to ascertain the status of the driver's vigilance.

[0070] A further example of means for testing the drivers response mayinclude means for flashing random numbers on the windscreen. The drivermay be prompted to respond by squeezing the steering wheel a number oftimes as determined by the number flashed. The numbers may be flashed onthe screen at different locations relative to the steering wheel withthe position of the hands on the wheel responding to the side of thescreen where the flashes were detected. This type of test should beconducted only when the driver is not turning, changing gear, braking orperforming other critical driving functions.

[0071] It is desirable to ensure that the driver response tests are notanticipated by the driver to more accurately detect the driver's stateof vigilance. It is of course also important that the selected method oftesting driver response, does not in any way distract the driver orcontribute to the driver's lapse in concentration.

[0072] The system may be built into a vehicle sun visor as a visualtouch screen display allowing a comprehensive visualisation of a driversactivity. The touch screen may include a color display for displayingmovement/pressure outputs associated with each sensor A display of thestatus of a plurality of sensors may provide a visual indication of arelaxed versus an active driver state.

[0073] According to one aspect of the present invention, there isprovided apparatus for determining a vigilance state of a subject suchas a driver of a vehicle or the like, said apparatus including:

[0074] means for monitoring one or more physiological variablesassociated with said subject;

[0075] means for deriving from said one or more variables datarepresenting physiological states of said subject corresponding to theor each variable; and

[0076] means for determining from said data when the vigilance state ofsaid subject is below a predetermined threshold.

[0077] According to a further aspect of the present invention, there isprovided a method for determining a vigilance state of a subject such asa driver of a vehicle or the like, said method including the steps of:

[0078] monitoring one or more physiological variables associated withsaid subject;

[0079] deriving from said one or more physiological variables datarepresenting physiological states of said subject corresponding to theor each variable; and

[0080] determining from said data when the vigilance state of saidsubject is below a predetermined threshold.

[0081] Preferred embodiments of the present invention will now bedescribed with reference to the accompanying drawings wherein.

[0082]FIG. 1 shows a block diagram of a vigilance monitoring systemaccording to the present invention;

[0083]FIG. 2 shows a flow diagram of an algorithm for processing datafrom sensors associated with a vehicle and driver;

[0084]FIG. 3A shows a simplified block diagram of a system forcancelling environmental noise from driver interfaced sensors;

[0085]FIG. 3B shows waveforms associated with the system of FIG. 3A;

[0086]FIG. 4A shows a flow diagram of a movement processing algorithmaccording to the present invention;

[0087]FIG. 4B shows examples of data reduction and syntactic signalprocessing associated with a sample signal waveform;

[0088]FIG. 5 shows sample outputs of data following secondary analysisby the system of FIG. 4A;

[0089]FIG. 6 shows an embodiment of steering wheels sensors;

[0090]FIG. 7 shows a block diagram of a vigilance monitoring systemutilizing video data:

[0091]FIG. 8 shows a flow diagram of an algorithm suitable forprocessing video data;

[0092]FIGS. 9 and 10 show examples of data produced by the system ofFIGS. 7 and 8;

[0093]FIG. 11 is a flow diagram of the main vigilance processingalgorithm;

[0094]FIG. 12 is a block diagram of a vehicle monitoring systemaccording to the present invention;

[0095]FIG. 13 shows one form of transducer for monitoring posture of adriver or equipment operator;

[0096]FIG. 14 shows a block diagram of an embodiment of an anti snoozedevice according to the present invention;

[0097]FIG. 15 shows a calibrate mode algorithm; and FIG.

[0098]FIG. 16 shows a main relax detection algorithm.

[0099] Referring to FIG. 1, block 12 shows a plurality of sensors 1 to11 associated with a vehicle and driver. The or each sensor may includepiezoelectric or electrostatic material such as CSD or PVDF material.The material can be divided into plural sections of the driver's seat,for example. The various sensors are summarized below.

[0100] 1. Upper Driver Seat Sensor

[0101] Drivers seat top edge of upper section

[0102] Drivers seat centre of upper section

[0103] Drivers seat base of upper section

[0104] 2. Lower Driver Seat Sensor

[0105] Drivers seat front edge of lower section

[0106] Drivers seat centre of lower section

[0107] Drivers seat rear of lower section

[0108] 3. Driver Seat-Belt Sensor

[0109] Driver's seat-belt upper section

[0110] Driver's seat-belt lower section

[0111] 4. Driver's Head Tilt Per Driver Cap or Similar Sensor

[0112] The driver's head tilt per driver cap or a device to clip overdrivers ear or as part of driving goggles or glasses. These sensors canbe, for example, positional switch devices. The output from thesepositional devices is amplified, filtered and finally data acquisitionedand analysed. This sensor device is designed to output a signal ordigital data which changes state in accordance with the tilt of thedriver's head. By calibration of the system in accordance with normaldriving conditions this output can correlate the normal drivingcondition with the fatigued driver condition.

[0113] 5. Driver Headband Sensor

[0114] The driver headband sensors can be, for example, a CapacitiveStatic Discharge Material (CSDM) or PVD Material (PVDM) that can bedivided into the various sections (as listed below) of the driver'sheadband sensor. The output from the various sensors is amplified,filtered and finally data acquisitioned and analysed. The headbandmaterial can contain conductive sections designed to pick-up thepatient's electro-encephalograph (EEG) signals.

[0115] Driver Headband Sensor

[0116] Driver headband left frontal

[0117] Driver headband right frontal

[0118] Driver headband left eye

[0119] Driver headband right eye

[0120] EEG, EMG and EOG parameters monitored in critical drivingconditions. In some critical applications of vigilance monitoring, suchas pilots of aircraft, personnel responsible for navigating andcontrolling ships, drivers of road or rail transport or passengervehicles, it can be appropriate to apply more comprehensive methods ofvigilance monitoring. These more comprehensive monitoring techniques caninclude techniques for analysing the frequency composition of a subjectsEEG physiological data. Half Period Amplitude analysis (AU patent632932) or spectral analysis can be applied in order to determine if thesubject is entering a trance or non-vigilant state or if the subject isbecoming drowsy. This type of sleep staging can be derived in real timeto facilitate determination of the subject's state of vigilance. If thesubject is detected as being in a risk category the present system willalert the driver in order to prevent a potential vehicle accident due tothe drivers lapse in concentration.

[0121] One method of electrode attachment, but not limited to, could bethe application of a headband by the driver where this head-band and/orchin-band could connect the EEG, EMG and EOG signals to the monitoringdevice for purpose of analysing the signals for determination of thesubjects state of wakefulness.

[0122] 6. Driver Eye Sensor

[0123] Various techniques can be applied for the purpose of eye movementmonitoring including;

[0124] Techniques based on reflected light.

[0125] Techniques based on electric skin potential.

[0126] Techniques based on Contact lenses

[0127] Techniques based on Limbus tracking

[0128] Techniques based on video imaging

[0129] Techniques based on Magnetic Induction

[0130] Driving goggles or glasses with infra-red detection capabilityfor monitoring driver's eye movements, or EOG signal pick up viaelectrodes.

[0131] Driver Eye Detection Sensor Types;

[0132] Driver's eyes left

[0133] Driver's eyes right

[0134] sources of eye movements can include EOG electrodes, infrareddetection of eye movements, or video tracking and processing of eyemovements.

[0135] 7. Driver Steering Wheel Sensor

[0136] The driver steering wheel or other steering device sensors can befor example, a CSDM or PVD material that can be divided into the varioussections (as listed below) of the driver's steering wheel or othersteering device. The output from the various sensors is amplified,filtered, and finally data acquisitioned and analysed.

[0137] Driver Steering Wheel Sensor Types;

[0138] Drivers steering wheel top left section

[0139] Drivers steering wheel top right section

[0140] Drivers steering wheel bottom left section

[0141] Drivers steering wheel bottom right section

[0142] An alternative form of steering wheel sensor is shown in FIG. 6.

[0143] 8. Driver Carpet Region Sensor

[0144] The driver carpet sensors can be, for example, a CapacitiveStatic Discharge Material (CSDM) or PVD Material (PVDM) that can bedivided into the various sections (as listed below) of the driver'scarpet area. The output from the various sensors is amplified, filteredand finally data acquisitioned and analysed.

[0145] 9. Driver Accelerator Sensor

[0146] The driver accelerator sensors can be, for example, a CapacitiveStatic Discharge Material (CSDM) or PVD Material (PVDM) that can bedivided into the various sections (as listed below) of the acceleratorpedal. The output from the various sensors is amplified, filtered andfinally data acquisitioned and analysed.

[0147] Driver Accelerator Pedal Sensor Types;

[0148] Drivers accelerator pedal top section

[0149] Drivers accelerator pedal center section

[0150] Drivers accelerator pedal bottom section

[0151] 10. Driver Clutch Pedal (where Applicable) Sensor

[0152] The driver clutch sensors can be, for example, a CapacitiveStatic Discharge Material (CSDM) or PVD Material (PVDM) that can bedivided into the various sections (as listed below) of the driver'sclutch pedal (where applicable). The output from the various sensors isamplified, filtered and finally data acquisitioned and analysed.

[0153] Driver Clutch Pedal Sensor Types;

[0154] Drivers clutch pedal (if applicable) top section:

[0155] Drivers clutch pedal (if applicable) center section

[0156] Drivers clutch pedal (if applicable) bottom section

[0157] 11. Driver Brake Pedal Sensor

[0158] The driver brake sensors can be, for example, a Capacitive StaticDischarge Material (CSDM) or PVD Material (PVDM) that can be dividedinto the various sections (as listed below) of the brake pedal. Theoutput from the various sensors is amplified, filtered and finally dataacquisitioned and analysed.

[0159] Brake Pedal Sensor Types;

[0160] Drivers brake pedal top section

[0161] Drivers brake pedal center section

[0162] Drivers brake pedal bottom section

[0163] Other sensors are referred to in block 13, including steeringwheel movement and direction sensors and sensors for detectingenvironmental noise and vibrations.

[0164] The outputs from the various sensors are amplified and filteredin block 14 in preparation for analog to digital conversion in block 15.The sensor signals are input in digital form to block 16. Block 16includes a central processing unit and one or more algorithms forprocessing the digital signals. Block 16 also makes use of the vigilanceprocessing algorithm(s) in block 17. The vigilance processingalgorithm(s) in block 17 are adapted to determine the status of thedriver state of vigilance, alertness or wakefulness. This status may beexpressed as a vigilance factor (0-100%). Upon detecting a vigilancefactor which is below an acceptable threshold, the central processingunit may alert the driver of the vehicle and/or other vehicles. Thedriver alert means may include:

[0165] To alert external drivers;

[0166] Flashing Hazard Lights

[0167] Sounding of siren

[0168] Internal vehicle driver alert systems;

[0169] Scent sprays which are designed to activate the drivers vigilancestate

[0170] Vibration modulation for driver—can include vibration of steeringwheel or floor area to alert driver

[0171] Vibration modulation for driver seat-belt

[0172] Vibration modulation for driver steering wheel

[0173] Audible alarm system at frequencies and durations or sequence ofdurations as tested be most effective in alerting the driver

[0174] Cabin bright lights designed to avoid driving hazard but testedfor improving driver vigilance

[0175] Upon detecting a vigilance factor which is below an acceptablethreshold, the central processing unit may intervene in the control ofthe vehicle. Vehicle intervention may enable the vehicle to be broughtinto a safe or safer status. Vehicle intervention may include speedrestriction or reduction or complete removal of fuel supply. In somecircumstances the accelerator pedal may need to be disabled, for examplewhen a driver has his foot depressed on the accelerator pedal and is inan unsafe or fatigued state.

[0176] Where a driver is detected as ignoring or not responding toresponse requests or appropriate acknowledgement that the driver is in avigilant state, the vehicle may have its horn or hazard flashing lightsactivated to warn other drivers, and/or have its fuel injectionde-activated, and/or speed reduced by gentle and controlled safebraking.

[0177] Where a driver is detected as suffering from fatigue and is notresponding to response tests, the vehicle may have its fuel supplyreduced, and/or its speed reduced by gentle and controlled safe braking,to a safe cruising speed. The driver may then be prompted again beforethe vehicle undergoes further intervention.

[0178] Another option for vehicle intervention is to provide a form ofignition override, as used in some alcohol based systems. In this typeof system the vehicle ignition or starting process may be inhibited byan inappropriate driver state which in the present case may bedrowsiness or excessive fatigue.

[0179] In many modern vehicles vehicle intervention options may beinstigated by an onboard computer or electronic interface eg. bycommunication with the speed controller or fuel injection logic. Thecomputer system, may include intelligence to arbitrate the mostappropriate intervention sequence or process to minimize risk to thevehicle driver or its passengers.

[0180]FIG. 2 shows a flow diagram of an algorithm for processing datafrom sensors associated with the vehicle and driver. Block 20 shows aplurality of arrows on the left representing data inputs from varioussensors associated with a vehicle, following conversion to digital data.The digital data is input to block 21 which determines whether the dataconforms to valid amplitude thresholds stored in block 22. Signalsbeyond the thresholds are classified as noise or artefact and areignored. The data is then input to block 23 which detects whether thedata conforms to valid time duration thresholds stored in block 24.Signals beyond the thresholds are classified as invalid and are ignored.The thresholds stored in blocks 22 and 24 are, for the purpose of thepresent embodiment, determined empirically from experimental trials. Thedata is then input to block 25 for signal compression. The role of block25 is to simplify further processing by presenting the data in aminimized form. This is done by syntactic processing whereby main datapoints only of the signals such as various peaks, troughs and zerocrossings or central points defining peaks of the signals are presentedfor further processing. The data is then input to block 26 where it iscategorized and summarized in terms of amplitude or power range, numberof movements per second or other epoch, and phase relationships betweenthe signals. The data may be displayed on tabular or graphical formand/or may be subjected to further automated processing to determinevigilance status.

[0181]FIG. 3A shows a block diagram of a system for removingenvironmental noise from driver interfaced sensors. Block 30 representsvarious sensors for monitoring driver movements and block 31 representssensors for monitoring environmental vibration and noise and vehicleartefacts.

[0182] Blocks 32 and 33 represent circuits for amplifying and filteringsignals from blocks 30 and 31 respectively. Block 34 represents analogueto digital converters for converting the signals from blocks 32 and 33into digital form for processing via the digital signal processor inblock 35. Block 35 includes an algorithm for performing signalcancellation as illustrated in FIG. 3B.

[0183] In FIG. 3B waveform A represents a signal from a driverinterfaced sensor or sensors (Block 30 of FIG. 3A). Waveform Brepresents a signal from a sensor or sensors associated with the vehicleengine and road noise pickup locations (Block 31 of FIG. 3A). Waveform Crepresents a signal after it is processed by Block 35. It may be seenthat the signal represented by waveform C is obtained by cancelling orsubtracting the signal represented by waveform B from the signalrepresented by waveform A. The signal represented by waveform C is atrue or valid movement signal which is not corrupted by environmentalnoise.

[0184]FIG. 4A shows a flow diagram of a movement processing algorithmaccording to the present invention. Referring to FIG. 4A, signals fromsensors 1 to 11 shown in block 12 of FIG. 1 are filtered, thenreferenced to period and amplitude threshold values before beingconverted to syntactic data. The syntactic data is correlated fordetermination of certain combinations of sensor movement signalsindicating that the driver is in a vigilant or wakeful state. When asensor signal or any combination of sensor signals are analysed as beingvoid of subject movement, this may be interpreted as an indication thedriver is suspected of being in a non-vigilant or fatigued state.Analysis of the fatigued state is determined by certain expectedpatterns from the various sensor signals. Such patterns include verylittle movement from the steering wheel and very little movement fromthe seat sensors, indicating that the driver may be excessively relaxedand subject to fatigue, or at risk of fatigue on-set. The functions ofblocks 40 to 61 are as follows:

[0185] Block 40

[0186]FIG. 1 shows how the analog signals from sensors 1 to 11 are:converted to a digital signal (FIG. 1, block 15); input to the centralprocessing unit (FIG. 1, block 16); and processed by a vigilanceprocessing algorithm (FIG. 1, block 17). The start of the algorithm inFIG. 4A represents the start of a process, which is repeated many timesfor each input sensor 1 to 11 (FIG. 4A shows the process for sensors 1,2, 3). This process analyses data from each input sensor for the purposeof final determination of the driver's vigilance state, and whether thisstate warrants an alarm alert in order to assist in preventing apotential accident.

[0187] Blocks 41 to 46

[0188] Signal A/D Data Output. The analog signal from each sensor isamplified, filtered and then converted to a digital signal inpreparation for signal processing.

[0189] Variables A,C,E-U

[0190] Variables A,C,E,-U provide to the processing algorithms thresholdamplitude and period values to allow sensor signal data reductions to bedetermined and to allow data reduction and syntactic signal processing.The variables (A,C,E-U) are determined via controlled studies fromexperimental and research data.

[0191]FIG. 4B shows examples of: (1) signals components which areignored due to being below a minimum amplitude threshold, (2) syntacticdata where the signal is represented by troughs and peaks of the signal,and (3) high frequency component being ignored due to being below aminimum period threshold. The latter recognizes relatively lowerfrequencies which are typically due to driver movements.

[0192] Inputs from Sensors 4 to 11, Subject to System Configuration

[0193] Input from each of the vehicles sensors is amplified, filteredand then analog to digital converted, in preparation for signalprocessing. This is performed by blocks similar to blocks 41 to 46.Inputs from more than 11 sensors can be catered for if required.

[0194] Block 47

[0195] A,C,E,-U

[0196] Variable data via default table (as determined by clinical dataand/or neuro node self learning and adjustment), resulting fromcustomisation to specific subject's driving characteristics and systemadaptation. Variables: B,D,F,-V. By comparing the sensor data to variousamplitude thresholds and pulse periods, it is possible to ignore datathat is likely to be noise or artefact and include data that isdistinguishable as movement data from a driver. The movement data isdistinguished by measuring the amplitude and period characteristics ofthe sensor signal. Movement data is also distinguished by comparingsignal patterns and characteristics of sensors to patterns andcharacteristics of typical driver's movements (as determined bycomparative data used for correlating against current data, this databeing derived from system self-learning and/or calibration processes.)

[0197] Block 48

[0198] Is peak to peak amplitude of sensor output greater than thresholdvariable A? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference A.

[0199] Block 49

[0200] Is peak to peak amplitude of sensor output greater than thresholdvariable C? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference C.

[0201] Block 50

[0202] Is peak to peak amplitude of sensor output greater than thresholdvariable E? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference E.

[0203] Block 51

[0204] Is peak to peak amplitude of sensor output greater than thresholdvariable B? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference B.

[0205] Block 52

[0206] Is peak to peak amplitude of sensor output greater than thresholdvariable D? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference D.

[0207] Block 53

[0208] Is peak to peak amplitude of sensor output greater than thresholdvariable F? Retain time reference and value of each signal excursion ofinput sensor exceeding amplitude reference F.

[0209] Storage of input sensors period and amplitude with time reference

[0210] The syntactic data from the full range of sensors is stored inrandom access memory for the purpose of processing and determination ofa subject's vigilant state. The syntactic data is also archived to allowpost analysis report and validation or review of driver fatigue andperformance. This can be particularly useful where truck drivers andother critical transport or passenger drivers are required to be checkedfor performance and vigilance compliance.

[0211] Block 54

[0212] Longer-term data storage is designed to log the driver's movementdata from each of the sensors. This stored data can be accessed at alater stage in order to review the driver's performance history inregards to movement analysis and subsequent vigilance.

[0213] Block 55

[0214] Short term direct access storage used for storing parameters suchas the past 10 minutes of syntactic data for each sensor channel. Inorder to correlate the various data from each sensor or channel andcompare this data combination to pre-defined sets of rules designed todescribe combinations of sensor outputs which are typical of driverfatigue conditions.

[0215] Block 56

[0216] Store syntactic representation of sensor signal exceedingthreshold A and B, with timer reference, amplitude and pulse width.

[0217] Block 57

[0218] Store syntactic representation of sensor signal exceedingthreshold C and D, with timer reference, amplitude and pulse width.

[0219] Block 58

[0220] Store syntactic representation of sensor signal exceedingthreshold E and F, with timer reference, amplitude and pulse width.

[0221] Block 59

[0222] Driver specific profile and calibration data can be stored forlater correlation reference. By correlating with various thresholds orreference conditions the system is able to determine interaction tosensors when a particular driver's conditions is similar to pre-storedreference characteristics. This comparative data is stored as data inlook up tables. The data can consist of frequency and/or amplitudecharacteristics for a range of driver states or alternatively the datacan consist of samples of data (with acceptable vanations to the samplesof data) that exist for a range of driver states.

[0223] Block 60

[0224] Vehicle output signals. These include steering wheel movements,direction of steering wheel movements, speed of vehicle, change of speedof vehicle, engine vibration and noise, road vibration and noise

[0225] By processing driver steering wheel adjustments and comparingthese adjustments with the various sensor signals and correlation ofvarious sensor signals, it is possible to determine the probability thatthe driver is in a state of fatigue and the degree of driver fatigue.The vehicle signals are also analysed in order to assist in noisecancellation (ie vehicle noise as opposed to driver movement) and moreaccurate identification of valid driver movements).

[0226] Block 61

[0227] Correlate all channels of sensor activity and determine if driverfatigue is a probability and what level of driver fatigue is detected.Look up table of specific driver calibration values and reference statesis used to determine actual driver state and level of fatigue of driver,along with probability of data accuracy. Standard reference data tablesand default values are also used for determination of driver fatigue.See sample R&K style histograms, movement histograms and tables showingweighted value of each sensor and actual movement detection from eachsensor to determine fatigue probability as a function of movementdetection with appropriate weighting.

[0228]FIG. 5 shows typical samples of processed data following secondaryanalysis for sensor signals 1 to 4. The data shows in graphical form thenumber of valid movements detected for each sensors 1 to 4 duringsuccessive time intervals n, n+1, n+2, . . . Tertiary analysis may beperformed on this data which would allow simple to view correlationbetween the various sensors. The samples shown in FIG. 5 demonstrate anexample (dotted line) where the various sensors all experience obviousmovement detection.

[0229] The steering wheel sensors shown in FIG. 6 are divided into eightsections as follows:

[0230] Top 62, top left 63, top right 64, left 65, right 66, bottom left67, bottom right 68 and bottom 69.

[0231] Sensors 62-69 are linked via eight cables to output pins 1 to 8respectively. A common connection to each sensor is linked by cables tooutput pin 9. Alternative configurations are possible with more or lesssensors and with the option of sensor arrays on both the upper and lowersurfaces of the steering wheel grip surface. The outputs represented bypins 1 to 9 are connected to analogue signal conditioning circuits andvia analogue to digital convertors to digital signal processing circuitsas described above.

[0232] It is desirable to measure pressure of a driver's hand or handson the steering wheel at all times. The pressure may be compared toprevious values and/or calibrated values to determine whether a patternof increased or decreased pressure reflects driver fatigue onset.

[0233] If the driver's state of consciousness or concentration changesdue to fatigue onset or the like, the system may calculate and deduce anappropriate point at which the driver should be alerted. The appropriatepoint may be determined from a combination of pre-calibrated data for aspecific driver and/or pre-programmed patterns, states or trends in thedata including relative and absolute pressure values obtained from a setor subset of vehicle sensors.

[0234]FIG. 7 shows a block diagram of a vigilance monitoring systemutilizing video data. Block 70 represents a video CCD (charge coupleddevice) camera which may be located on the driver's visor, dash-board orother suitable location to enable video monitoring of the driver's eyes.An infra-red lens may be utilized to facilitate reliable night videomonitoring capability. The output of the video camera is passed to block71. Block 71 is an analog to digital converter for digitizing the videosignal prior to processing via block 72. Block 72 is a centralprocessing unit and includes a video processing algorithm. The videoprocessing algorithm has eye recognition software designed to identifyeyes in contrast to other parts of the drivers face. Eyes are detectedusing special processing software that allows the driver's eyes to beanalysed. This analysis includes determining the area of the eye'sopening and correlating the eye's opening area to previous similarmeasurements. In this way eye processing can determine whether adriver's eyes are remaining open as would be expected in an alert stateor whether the current eye opening of the driver is relatively less(when compared to earlier eye opening measurements). Rates or degrees ofeye closure are able to be detected and continually monitored in thismanner.

[0235] The video processing algorithm also detects blink rate andpossibly eye movements to determine whether the drivers eyes appear tobe alert or possibly fixed in a dangerous “trance state” as may beapparent during lapses of driver vigilance. Block 73 represents outputsof block 72 including

[0236] eyes blink rate

[0237] eyes closure, calculated as a percentage ratio of current eyesopen area to previously calculated maximal eyes open area.

[0238] eyes focus factor, determined by measuring number of eyemovements per second, extent of eye movements (ie small eye movements orlarger eye movement deflections)

[0239] the nature of eye movements can reflect appropriate patterns ofmovement of a driver's eyes such as focus on sections of the road for anappropriate time as well as inappropriate patterns of movementassociated with fatigue or lack of vigilance

[0240] type of eye movements, ie vertical, horizontal, stare

[0241] The above measures may be gauged against actual trials in orderto determine relevant indices that correlate to a driver's fatiguedstate.

[0242]FIG. 8 shows a flow diagram of an algorithm suitable forprocessing video data. The functions of blocks 80 to 94 are as follows:

[0243] Block 80

[0244] Start

[0245] Block 81

[0246] CAPTURE EYE VIDEO DATA—Capture current video frame. Digitisevideo frame of subject's eyes. Eye data can be captured via one or moreof the following means:

[0247] CCD video camera, Electro-oculogram data capture means viasubject worn headband, direct electrode attachment, driver glasses,head-cap or movement sensors, infrared or other light beam detectionmeans.

[0248] Block 82

[0249] Apply Eye Data Processing and Determine Left & Right Eye OpeningArea and Blink Events.

[0250] Apply edge detection, signal contrast variation and shaperecognition, amongst other processing techniques to determine the borderof the subject's eye lids. Determine area of each of the subject's eyeopenings, height of each eye opening, blink events for each eye, blinkrate and time reference associated with each blink event.

[0251] Block 83

[0252] Correlate Current and Past Video Captured Eye Movement Data

[0253] Correlate current eye position data with previous position eyedata. Review eye position trend data and determine trends and patternsof eye movements that indicate on-set of or driver fatigue state.Patterns include:

[0254] states of staring or trance like states indicating loss of roadconcentration.

[0255] slowly rolling eye movements (typical of sleep onset).

[0256] eye focus directions and association of these directions withdriver fatigue

[0257] Process digitised video frame and detect subject's left and righteye movement patterns and activity of eyes and association of thisactivity with driver fatigue.

[0258] Compare current blink rates, past blink rates and look-up tableblink rate characteristics, thresholds for various fatigue on-set andfatigue blink rates and blink characteristics associated with variousdriver states.

[0259] Compare current eye opening area with thresholds for fatigue andfatigue on-set conditions to determine vigilant driver eye openingstatus versus fatigued driver eye opening status.

[0260] Block 84

[0261] Look up table with characteristic patterns of;

[0262] eye movements and threshold data for fatigued versus vigilantsubjects.

[0263] Blink rate typical thresholds and characteristics

[0264] Eye opening typical and default thresholds

[0265] Eye movement typical and default characteristics for driverfatigue on-set.

[0266] Block 85

[0267] Store subject's left & right eye opening area, eye openingheight, blink rates, eye position and eye movements together with timereference.

[0268] Block 86

[0269] Calibration data derived from subject and vehicle calibrationprocedures.

[0270] determination of fatigue on-set blink rates thresholds.

[0271] Determination of eye opening fatigue on-set thresholds.

[0272] Determination of eye position, movement characteristics andactivity characteristics for fatigue on-set thresholds.

[0273] EOG patterns for wake, drive activity, fatigue on-set, fatigue.

[0274] Trance and hypnotic EOG eye characteristics.

[0275] Block 87

[0276] Fatigue threshold time period variable X set from default values,subject calibration or system self-learning/calculation.

[0277] Block 88

[0278] Is mean of eye opening area below “fatigue mean eye openingthreshold X”?

[0279] Block 89

[0280] Fatigue threshold time period variable Y set from default values,subject calibration or system self-learning/calculation.

[0281] Block 90

[0282] Is time duration below mean eye opening fatigue threshold (X)greater than Y?

[0283] Block 91

[0284] Blink rate fatigue characteristics set from default values,subject calibration or system self-learning/calculation.

[0285] Block 92

[0286] Does blink rate and characteristics comply with fatigue blinkrate characteristics?

[0287] Block 93

[0288] Apply eye data processing and determine left & right opening areaand blink events.

[0289] Correlate current and past video captured eye movement data.

[0290] Detection of fatigue eye opening on-set and detection of fatigueblink rate on-set.

[0291] Block 94

[0292] Eye movement fatigue determination diagram.

[0293]FIGS. 9 and 10 show examples of eye opening and eye position dataproduced by the system of FIGS. 7 and 8.

[0294]FIG. 11 is a flow chart of the main vigilance processingalgorithm. The functions of blocks 95 to 99 are as follows:

[0295] Block 95

[0296] Main Vigilance Processing Algorithn

[0297] Vigilance Movement Processing Algorithm. (see FIG. 4A)

[0298] Vigilance Eye Status Processing Algorithm.

[0299] Probability of Driver Fatigue and Degree of VigilanceDetermination Algorithm (correlates subject Movement Status and EyeProcessing Status).

[0300] Block 96

[0301] LED indicator display panel.

[0302] Block 97

[0303] Eye Status Vigilance factor 0-100%.

[0304] Block 98

[0305] Movement Vigilance Factor

[0306] 0-100%—displayed as bar graph, meter or other means.

[0307] Block 99

[0308] Vigilance probability Factor 0-100%

[0309]FIG. 12 is a block diagram of a vehicle monitoring systemaccording to the present invention. FIG. 12 is an overview of a systemwhich utilizes many of the features discussed herein. The functions ofblocks 100 to 118 are as follows:

[0310] Block 100

[0311] Driver EEG sensors—direct attach electrode, headband, wirelesselectrode, driver cap and other EEG signal pick-up means.

[0312] Block 101

[0313] Driver EEG sensors—direct attach electrode, headband, wirelesselectrode, driver cap and other EEG signal pickup means.

[0314] Block 102

[0315] Driver Motion, Movement and Physiological Parameter sensors.

[0316] Block 103

[0317] Driver Eye Movement

[0318] Detection via electrode, driver glasses/goggles, infrared orother light beam means of tracking detection or other means.

[0319] Block 104

[0320] Vehicle status interface; speed, direction, accelerator position,break position, indicators, lights amongst other vehicle status data.

[0321] Block 105

[0322] In phase signal detection and processing. Applies processingwhich determines patterns of in-phase signal occurrence and associatesthese with driver or background noise as originating source.

[0323] Block 106

[0324] Anti-phase signal detection and processing. Applies processingwhich determines patterns of anti-phase signal occurrence and associatesthese with driver or background noise as originating source.

[0325] Block 107

[0326] Vehicle Background Noise Processing Algorithm.

[0327] Vehicle background and Environmental Noise Sensors to allow noisecancellation, filtering and reduction.

[0328] These sensors include microphone and vibration sensors located atstrategic positions in order to pick up background vehicle noise such asroad noise and engine noise. Fourier transform and frequency analysis ofbackground noise assists in selection of digital filteringcharacteristics to most effectively minimise vehicle environmental noiseand assist in distinguishing driver related fatigue monitoring signals.System will continually “self-leam” various vehicle background andthreshold noise levels, frequency and other characteristics in order todetermine changing vehicle noise conditions and subsequent noisecancellation or capability to ignore unwanted vehicle noise whileprocessing “real” driver movement and physiological signals andsubsequent fatigue status.

[0329] Artificial Intelligence:

[0330] Signal characteristics as generated by a range of varying roadconditions can be programmed into the system. The input data relating tovarious road conditions thereby provides a means to further distinguishwanted driver related signals from unwanted background noise signals.

[0331] Block 108

[0332] Driver EEG Sensors—Direct Attach Electrode Algorithm

[0333] Block 109

[0334] Driver EEG sensors—Direct Attach Electrode, Algorithm

[0335] Block 110

[0336] Driver Motion, Movement, Physiology Algorithm

[0337] Block 111

[0338] Driver Eye Movement Detection Algorithm

[0339] Block 112

[0340] Vehicle Status Interface Algorithm

[0341] Block 113

[0342] Driver Fatigue Processing Algorithm. Correlation with previousdriver fatigue conditions together with comparison of outputs for eachof above listed fatigue algorithms (Driver EEG, motion, eye, vehiclestatus).

[0343] Block 114

[0344] Driver vigilance interactive response testing.

[0345] Block 115

[0346] Driver alert and alarm systems for re-instatement of vigilance.

[0347] Block 116

[0348] Driver vehicle car intervention to reduce or limit speed andother means of increasing vehicle safety and reducing vulnerability todriver fatigue status.

[0349] Block 117

[0350] Vehicle fatigue display systems for displaying to the driver thecurrent fatigue status or early warning indicators of fatigue status.

[0351] Block 118

[0352] System communication storage and printing peripheral interface.Data storage, reporting processing, reporting print interface, wirelessand wire connected interfaces, for real-time or post communication offatigue data and fatigue status information. System can include GSM,cellular phone, satellite or other means of moving vehicle tracking anddata exchange in real-time or at any required later stage. Thisinformation transfer can be an effective means for trucks and othervehicles to have their driver status processed and reviewed, asappropriate and as required.

[0353]FIG. 13 shows one form of transducer for monitoring posture of adriver or equipment operator. FIG. 13 shows a webbed structurecomprising strips or elements of flexible PVDF or Piezo materialseparated by flexible insulation material terminated at A, B, C, D, E,F, G and H. Output signals from the respective strips are buffered,amplified, filtered and then analog to digital converted to data. Thisdata may be processed to determine an actual position of pressureapplied to the above structure. By analysing the two main co-ordinatesand the amplitudes of signals associated with those co-ordinates, theexact position of pressure applied by the vehicle driver or equipmentoperator may be determined.

[0354] The position where greatest pressure is applied is defined by theintersection of web strip pairs (eg. Band F) which produce the greatestsignal amplitude. The position may be described by coordinatesreflecting the web strip pairs (eg. B,F) which produce the greatestsignal amplitude. The above transducer may be used in conjunction withthe movement sensors described herein to provide a further layer ofpositional information relating to applied pressure for each sensor.This information may be important in circumstances where a driver'spressure to the steering wheel or the driver's pattern of hand placement(with respective applied pressure) varies in accordance with alertnessand drowsiness.

[0355] The posture of the driver or equipment operator may be monitored,stored, correlated with various threshold states and/or displayed inmeaningful graphic or numerical form. The threshold states may bederived by way of calibration for each specific driver's posture profileunder various states of fatigue and/or stress states and conditions.

[0356] The anti-snooze device shown in FIG. 14 includes sensors (block120) connected to an acquisition and processing means (block 121). Block122 includes monitoring means designed to amplify, filter and digital toanalog convert driver sensor signals in preparation for digital signalprocessing. The digital signal processing means (block 121) includes acalibration algorithm as shown in FIG. 15 and a main relax detectionalgorithm as shown in FIG. 16.

[0357] The driver can select the relax calibration function, then takeon the driving posture that would most closely represents a relaxed orpossibly fatigued driving state and the system will then monitor andstore the minimum threshold of driver activity over a period ofapproximately but not limited to 10 seconds, as a relaxed driverreference level.

[0358] The driver can select an active calibration function, then takeon the driving posture that would most closely represents normal drivingstate and the system will then monitor and store the minimum thresholdof driver activity over a period of approximately but not limited to 10seconds, as an active driver reference level.

[0359] The relaxed and active driver reference levels stored in thesystem may be displayed on the visual touch screen display for varioussensors. The system may perform a validation function by replaying thedrivers relaxed and active reference levels on the touch screen. Thisallows easy comparison to be made with actual sensor levels when thedriver adopts postures representing normal/fatigued states and serves tovalidate the correctness of the stored reference levels.

[0360] The driver can also select a sensitivity function which maydetermine how close to the relaxed level the driver needs to be beforethe anti-snooze system alerts the driver. By viewing the anti-snoozedevice screen the driver can relax or adopt normal vigilant drivingposture and adjust sensitivity control so that the anti-snooze deviceappears to track and detect the drivers relaxed state. The anti-snoozedevice has the ability to act as a self warning aid by simply alertingthe driver when his posture or driving vigilance is deteriorating. If,for example, a driver's steering wheel grip erodes or undergoes fatigue,the anti-snooze system can be calibrated to detect this condition andalert the driver.

[0361] It is possible for the driver to have calibration data determinedby an off-road simulator that more accurately defines thecharacteristics of each specific drivers activity variations andphysiological variations during dangerously relaxed or fatigued drivingconditions. The calibration data can be up-loaded to the anti-snoozedevice to provide more accurate relaxed and active reference levels. Thecalibration data may also provide more accurate means of determining therelative effect that each individual sensor has during a driverstransition from active and alert to drowsy and fatigued. The effects ofeach sensor may be recorded and this data may assist in more accurateanti-snooze detection.

[0362] During calibration modes the system may detect the drivers handpressures via the steering wheel sensors, the drivers respiration andECG via the seatbelt sensors, and the drivers posture and movement viathe seat sensors.

[0363] The anti-snooze system may continually monitor and average thesignal amplitudes of all sensors, while comparing the current levels ofsensor amplitude with the calibrated levels. The system may also comparecurrent movement sensor patterns to reference data. This reference datacan represent certain threshold levels calibrated to each individualdriver or general reference conditions. The various sensors may beweighted in accordance with their respective importance in determiningwhether a driver's current state of activity is below the threshold orappropriately close to the relaxed mode calibrated reference level towarrant that the driver be alerted.

[0364] If the driver is detected as being within the range of sensoramplitudes and activity to warrant being alerted, the anti-snooze devicecan restrict the speed of the vehicle or slowly bring the vehicle to astand still in order to reduce the likelihood of an accident. Thisability to restrict the vehicle's speed could be overridden by thedriver as is possible in “auto-cruise” devices currently available onmany vehicles.

[0365] The techniques and methodologies may include relatively complexneurological waveform analysis techniques, video tracking of driver eyemotions, sophisticated noise cancellation and simpler driver interactiveprocesses such as sensitizing the steering wheel, seat-belt, gear-stickand other driver cabin regions.

[0366] One application for the present invention may include a truckdriver vigilance monitoring (TDVM) system. This system may be designedaround the “dead-man” handle concept as applied successfully in trains.A variation of this system may provide visual cues and driver vigilanceresponse testing.

[0367] The TDVM system may include pre-programmed Light Emitting Diode(LED) displays to be activated in various sequences and at variousfrequencies and durations. The truck driver can be visually prompted byway of these LEDS to press the steering wheel according to whether theleft or right or both LEDS are flashed. The response time and accuracyof the driver's response to the prompts may be measured and relayed backto a remote monitoring control station.

[0368] Various drivers will have calibrated “vigilant response times andaccuracy levels” which can be compared to actual current response times.Where appropriate, an alarm can be activated, if the response timesindicate fatigue on-set or a potentially dangerous state.

[0369] The sequences and durations can be validated in accordance withclinical trials to provide an effective method of vigilance detection.Sequences and patterns of visual truck cabin prompts can be establishedto minimize driver conditioning. Frequency of vigilance test prompts canbe determined in accordance with requirements as determined via fieldstudies.

[0370] Safety considerations to avoid driver distraction by the proposedmonitoring system may be implemented. Techniques such as utilization of“busy” response prompts especially designed within the system to alertthe monitoring control unit that the driver is vigilant but unable torespond at the time due to driving demands.

[0371] The TDVM system may include the following components:

[0372] 1. Analysis Software This software may include a processingalgorithm(s) designed to evaluate various driver prompts and responsetimes. Evaluation of these response times may produce a probabilityfactor associated with driver vigilance for each specific driver.Analysis capability of driver response times may be an important elementof the system. Accuracy of vigilance probability outcome, clinicalanalysis and scientific validation associated with this process maydetermine effectiveness of the monitoring system.

[0373] 2. Truck-Cabin Steering-Wheel Physiological Movement Transducer.

[0374] This device may adapt to the truck steering wheel and provideoutput signals subject to a particular zone of the steering wheel, whichhas been activated by applying various degrees of pressure to thesteering wheel.

[0375] 3. Controller Unit & Monitoring Device (CU&MD).

[0376] This device may provide a communication link and data managementfor interfacing the truck's CU&MD to a remotely located monitoringstation.

[0377] This device may also provide the transducer interface andtransducer signal recording and detection capabilities.

[0378] This device may also output control to the driver indicator LEDSand record and transmit vigilance response times to the remotemonitoring station.

[0379] 4. Vigilance LED Display.

[0380] This device may be interfaced to the CU&MD unit and may providevisual response prompt to the truck driver.

[0381] 5. Remote Recording, Monitoring and Analysis System.

[0382] This system may facilitate a remote operators visual alarms whenvigilance response times are outside acceptable thresholds.

[0383] This system may also provide communication links to the truck.

[0384] This system may also provide analysis and system reporting toallow real-time tracking of vigilance performance and vigilance alarmstatus.

[0385] Finally, it is to be understood that various alterations,modifications and/or additions may be introduced into the constructionsand arrangements of parts previously described without departing fromthe spirit or ambit of the invention.

1. Apparatus for determining a vigilance state of a subject such as adriver of a vehicle or the like, said apparatus including: means formonitoring one or more physiological variables associated with saidsubject; means for deriving from said one or more variables datarepresenting physiological states of said subject corresponding to theor each variable; and means for determining from said data when thevigilance state of said subject is below a predetermined threshold. 2.Apparatus according to claim 1 wherein said physiological variablesinclude body movement and/or position, EEG, EMG & EOG signals,electrical skin resistance and eye and head movement and/or position. 3.Apparatus according to claim 1 or claim 2 wherein said monitoring meansincludes at least one sensor placed on the subject or a zone associatedwith the vehicle.
 4. Apparatus according to claim 3 wherein the or eachsensor includes CSD or PVDF material.
 5. Apparatus according to claim 4wherein the sensor material is placed on a steering wheel, seat, seatbelt and/or pedals of the vehicle.
 6. Apparatus according to claim 5wherein the pedals include an accelerator, clutch and brake pedals ofthe vehicle.
 7. Apparatus according to claim 1 wherein at least one ofsaid means for deriving and said means for determining is provided viadigital processing means.
 8. Apparatus according to any one of thepreceding claims wherein said determining means includes a vigilancealgorithm.
 9. Apparatus according to claim 8 wherein said vigilancealgorithm is adapted to correlate said data and/or apply combinationallogic to said data to detect patterns in said data which are associatedwith a vigilance state of the subject that is below said predeterminedthreshold.
 10. Apparatus according to claim 9 wherein said algorithmincorporates one or more look up tables including reference movementdata and default states associated with vigilance states that are aboveand below said predetermined threshold.
 11. Apparatus according to claim8 wherein said algorithm is adapted to determine a vigilance probabilityfactor as a function of weighted movement data values.
 12. Apparatusaccording to any one of the preceding claims including means forintervening with the control of the vehicle and/or alerting the subject,in the event that the vigilance state of said subject is below saidpredetermined threshold.
 13. Apparatus according to claim 12 whereinsaid means for intervening includes controlled application of brakes,cutting-off fuel supply and/or disabling the accelerator pedal of saidvehicle.
 14. Apparatus according to claim 12 wherein said means foralerting includes at least one of a spray designed to stimulate thesubject, vibrating a steering wheel, seat belt or floor area associatedwith the vehicle; an audible alarm and/or switching of bright lights inthe cabin of the vehicle, and flashing hazard lights and/or sounding ofan audible alarm such as siren or horn.
 15. Apparatus according to claim12 wherein said means for intervening is interfaced to a remotemonitoring station via a wireless communication system.
 16. A method fordetermining a vigilance state of a subject such as a driver of a vehicleor the like, said method including the steps of: monitoring one or morephysiological variables associated with said subject; deriving from saidone or more physiological variables data representing physiologicalstates of said subject corresponding to the or each variable; anddetermining from said data when the vigilance state of said subject isbelow a predetermined threshold.
 17. A method according to claim 16wherein said physiological variables include body movement and/orposition, EEG, EMG & EOG signals, electrical skin resistance and eye andhead movement and/or position.
 18. A method according to claim 16 orclaim 17 wherein said step of monitoring includes placing at least onesensor on the subject or a zone associated with the vehicle.
 19. Amethod according to claim 18 wherein the or each sensor includes CSD orPVDF material.
 20. A method according to claim 19 wherein the sensormaterial is placed on a steering wheel, seat, seat belt and/or pedals ofthe vehicle.
 21. A method according to claim 20 wherein the pedalsinclude an accelerator, clutch and brake pedals of the vehicle.
 22. Amethod according to claim 16 wherein at least one of said step ofderiving and said step of determining is performed via digitalprocessing means.
 23. A method according to any one of claims 16 to 22wherein said step of determining is performed via a vigilance algorithm.24. A method according to claim 23 wherein said vigilance algorithm isadapted to correlate said data and/or apply combinational logic to saiddata to detect patterns in said data which are associated with avigilance state of the subject that is below said predeterminedthreshold.
 25. A method according to claim 24 wherein said algorithmincorporates one or more look up tables including reference movementdata and default states associated with vigilance states that are aboveand below said predetermined threshold.
 26. A method according to claim23 wherein said algorithm is adapted to determine a vigilanceprobability factor as a function of weighted movement data values.
 27. Amethod according to any one of claims 16 to 26 including the step ofintervening with the control of the vehicle and/or alerting the subject,in the event that the vigilance state of said subject is below saidpredetermined threshold.
 28. A method according to claim 27 wherein thestep of intervening includes controlled application of brakes,cutting-off fuel supply and/or disabling the accelerator pedal of saidvehicle.
 29. A method according to claim 27 wherein the step of alertingincludes at least one of applying a spray designed to stimulate thesubject, vibrating a steering wheel, seat belt or floor area associatedwith the vehicle; activating an audible alarm and/or switching of brightlights in the cabin of the vehicle, and flashing hazard lights and/orsounding of an audible alarm such as siren or horn.
 30. A methodaccording to claim 27 wherein the step of intervening includesinterfacing to a remote monitoring station via a wireless communicationsystem.
 31. Apparatus for determining vigilance state of a subjectsubstantially as herein described with reference to any one of theembodiments illustrated in the accompanying drawings.
 32. A method fordetermining a vigilance state of a subject substantially as hereindescribed with reference to any one of the embodiments illustrated inthe accompanying drawings.